https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=Ryr5Sej5R1ekspHn3UTtbA.0&searchStr=linux%2C18.04%2Cx64%2Cdebug%2Cmochitests%2Ctest-linux1804-64%2Fdebug-mochitest-devtools-chrome-e10s%2Cdt4&tochange=c6acf0da4833498d9cc1d9587ca80296d928e27e&fromchange=aa164511257356798613bcbb939b73e5dee38d52

is a devtools chunk which fails.

Log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=317721651&repo=autoland&lineNumber=2031
[task 2020-10-06T06:40:32.389Z] 06:40:32 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtests.py', '--total-chunks', '12', '--this-chunk', '4', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--appname=/builds/worker/workspace/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=/builds/worker/workspace/build/symbols', '--certificate-path=tests/certs', '--setpref=webgl.force-enabled=true', '--quiet', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-devtools-chrome_raw.log', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-devtools-chrome_errorsummary.log', '--use-test-media-devices', '--screenshot-on-fail', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=/builds/worker/workspace/build', '--log-raw=-', '--flavor=browser', '--subsuite=devtools', '--chunk-by-runtime'] with output_timeout 1000
[task 2020-10-06T06:40:32.662Z] 06:40:32 INFO - Checking for ssltunnel processes...
[task 2020-10-06T06:40:32.669Z] 06:40:32 INFO - Checking for xpcshell processes...
[task 2020-10-06T06:40:32.773Z] 06:40:32 INFO - Traceback (most recent call last):
[task 2020-10-06T06:40:32.775Z] 06:40:32 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 3320, in <module>
[task 2020-10-06T06:40:32.775Z] 06:40:32 INFO - sys.exit(cli())
[task 2020-10-06T06:40:32.776Z] 06:40:32 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 3316, in cli
[task 2020-10-06T06:40:32.777Z] 06:40:32 INFO - return run_test_harness(parser, options)
[task 2020-10-06T06:40:32.778Z] 06:40:32 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 3301, in run_test_harness
[task 2020-10-06T06:40:32.778Z] 06:40:32 INFO - result = runner.runTests(options)
[task 2020-10-06T06:40:32.779Z] 06:40:32 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2697, in runTests
[task 2020-10-06T06:40:32.780Z] 06:40:32 INFO - tests = self.getActiveTests(options)
[task 2020-10-06T06:40:32.780Z] 06:40:32 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 1514, in getActiveTests
[task 2020-10-06T06:40:32.781Z] 06:40:32 INFO - runtimes = json.load(f).get(options.suite_name, {})
[task 2020-10-06T06:40:32.782Z] 06:40:32 INFO - AttributeError: 'Namespace' object has no attribute 'suite_name'

The log shows the variable MOZHARNESS_TEST_PATHS which holds the test manifests which shall be executed is actually missing.

Odd, that shouldn't be possible! Will take a look when I have a chance.

For a quick update, I didn't forget about this and I have dug into this at various points. It's likely a regression from that change where we made sure that tasks appear in the full taskgraph, even when no manifests are selected so that they're available for the backfill.

Though for some reason the task isn't being removed at the target task or optimization phase. The fix should be easy, but I wanted to understand why this was happening before implementing it. That's the part I haven't figured out yet.

Sorry I didn't get to this prior to my leave. I decided to focus on other bugs due to the relatively low intermittency rate here. This is still on the radar though.

options.suite_name was added in https://hg.mozilla.org/mozilla-central/rev/7ce1fc0e8368 (bug 1637463), but only when run from mach. It's not obvious to me why the missing options.suite_name is not causing more trouble in mozharness-initiated tasks chunked by runtime.

(In reply to Geoff Brown [:gbrown] from comment #9)

It's not obvious to me why the missing options.suite_name is not causing more trouble in mozharness-initiated tasks chunked by runtime.

I see now: Most (all?) mozharness-initiated mochitests run with --chunk-by-runtime and specify a list of manifests, rather than using --total-chunks and --this-chunk. Then this branch is never taken:

https://searchfox.org/mozilla-central/rev/71621bfa47a371f2b1ccfd33c704913124afb933/testing/mochitest/runtests.py#1563

So the missing suite_name issue is only encountered when both --total-chunks and --chunk-by-runtime are specified, and that's not expected in the current run-by-specified-manifests environment.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.